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Abstract — Here we are taking database of 264 sketch-photo 
pairs consisting of 50 forensic images, we are proposing a 
forensic sketch recognition system in which we are going to use 
SIFT and MLBP algorithms for feature extraction with an 
LFDA framework for dimensionality reduction. SIFT and 
MLBP are used for feature extraction. Training is applied to all 
sketches and photos in database. We will calculate the accuracy 
of matched sketch and photo pair by its rank. Rank 1 will be the 
maximum matched features and Rank 50 will be least matched 
sketch 


Index Terms — Forensic sketches, LFDA (Local Feature 
Discriminant Analysis), SIFT (Scale Invariant Feature 
Transform), MLBP (Multiscale Local Binary pattern). 

I. Introduction 

Today, advances in biometric technology have provided 
criminal investigators additional tools to help determine the 
identity of criminals. In addition to the evidence, if a dormant 
fingerprint is found at the scene of crime or a surveillance 
camera captures an image of the face of a suspect, then these 
clues are used in finding the suspect using biometric 
identification techniques. However, many crimes occur where 
not a single of the above discussed information is present. 
Also, the lack of technology to effectively capture the 
biometric data like finger prints within a short span after the 
scene of crime is a routine problem in remote areas. Despite 
these , many a times, an eyewitness account of the crime is 
available who had seen the criminal. The Police patrol 
deploys a forensic artist to work with the witness in order to 
draw a sketch that sketches the facial appearance of the 
culprit. These sketches are known as forensic sketches. At 
Once the sketch is ready, it is sent to the law enforcement 
officers and media outlets with the hope of catch holding the 
suspect. Here, two different conditions may arise for the 
culprit: 

1 . The person may have already been convicted once or 

2. The person has not been convicted even once or this is the 
first time, he may be committing crime. 

Here we deal with the first type of the scenario. If the criminal 
has been convicted at-least once, a mug shot photo (photo 
taken, while the person is being sent to jail) is available. Using 
an efficient forensic sketch matching system, the police can 
narrow down the potential suspects which will reduce the 
future crimes by the same criminal drastically. 

We are using 2 types of sketches; Viewed sketches are 


sketches that are drawn while viewing a photograph of the 
person or the person himself. Forensic sketches are drawn by 
interviewing a witness to gain a description of the suspect. 

II. LITERATURE SURVEY 

Research in sketch matching started only a decade ago. This 
is because the accuracy of sketch recognition is very low, 
compared to traditional face recognition techniques. This is in 
turn due to a large texture difference, between a sketch and a 
photo. Since researchers struggled to get good results. 

Most of the work in matching viewed sketches was performed 
by Tang and Wang [1] [2]. Tang and Wang first approached 
the problem using an eigentransformation method [1] to either 
project a sketch image into a photo subspace, or to project a 
photo image into a sketch subspace. Once projected into the 
same image subspace, they were matched using a PCA-based 
matcher. An improvement to this method was offered by 
Wang and Tang [2], where the relationship between sketch 
and photo image patches was modeled with a Markov random 
field. Belief propagation was used to minimize the energy 
between the selected patches and their corresponding sketch 
or photo mates as well as their selected neighboring patches. 
Here, the synthetic sketches generated were matched to a 
gallery of photographs using a variety of standard face 
recognition algorithms. 

In the paper[3] the authors discussed a method for 
representing face which is based on the features which uses 
geometric relationship among the facial features like mouth, 
nose and eyes .Feature based face representation is done by 
independently matching templates of three facial regions i.e. 
eyes, mouth and nose .Principal Component Analysis method 
which is also called Eigen faces is appearance based 
technique used widely for the dimensionality reduction and 
recorded a greater performance in face recognition. Thus they 
discussed about principle component analysis (PCA) 
followed by Feed Forward Neural Network called principal 
component analysis-neural network (PCA-NN). 

. In paper [4] which presents a novel and efficient facial image 
representation basebd on local binary pattern (LBP) texture 
features. The face image is divided into several regions from 
which the LBP feature distributions are extracted and 
concatenated into an enhanced feature vector to be used as a 
face descriptor. To identify forensic sketches much efficient 
algorithm is presented here in [5]. Both sketches and photos 
are considered for extracting feature descriptors using Scale 
Invariant Feature Transform corresponding (SIFT). The goal 
is to build a system that accurately matches forensic sketches 
with their photo images using feature based approach 
The authors in paper [6] discussed recent developments in 
automated face recognition that impact the forensic face 
recognition community. Improvements in forensic face 
recognition through research in facial aging, facial marks, 
forensic sketch recognition, face recognition in video, 
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near-infrared face recognition, and use of soft biometrics are 
discussed In the paper [7], authors compared the 
performances of humans and a principle component analysis 
(PC A) -based algorithm in recognizing face sketches. The 
experiments were carried out by matching sketches in a probe 
set to photographs in a gallery set. 

III. PROBLEM STATEMENT 

The rapid development in computer graphics, realistic 
animations, human computer interaction and biometric 
technology has provided law enforcement agencies with 
additional tools to determine the identity of the suspects. In 
most of the cases, the photo image of a suspect is not 
available. The best substitute available is always an artist 
drawing based on the recollection of an eyewitness. These 
sketches are called as forensic sketches which can effectively 
help investigators to locate or narrow down the potential 
suspects. 

The major challenge of face sketch recognition is matching 
images of different modalities. Basically, a face photo is 
captured by a digital camera, while a face sketch is drawn by 
artist with different level of information. Even for the same 
human subject, the face photo and its sketch might be 
different. The face shape might be exaggerated by artist or 
texture might be lost or replaced by some artistic rendering. 
This problem will be more exacerbated for forensic 
investigations, when the eye-witness cannot exactly 
remember the detail of suspect’s face. 

Earlier mentioned approaches in the literature are mainly based 
on matching viewed sketches that are highly accurate sketches 
which are drawn while looking at the subject. So, forensic 
sketches differ from viewed sketches. Hence, the proposed 
work focuses on matching forensic sketches to a large gallery 
of mugshot image database. 

IV. PROPOSED SYSTEM 

Most previously discussed methods from literature survey for 
matching viewed sketches and photos are based on either 
performing a global conversion from one domain to the other, 
or by using a patch-based conversion that replaces each 
sketch image patch by a photo patch in a training set. These 
methods have over published algorithms using the proposed 
the advantage of being able to generate a synthetic 
photograph of a sketch or vice-versa, which can then be 
matched using existing face recognition algorithms. However, 
the performance of these methods is limited because they are 
dependent on the amount of training data. When removing the 
constraints provided by a closed data set, these methods are 
subject to errors in estimating the synthetic images due to the 
large dimensionality and range of pixel values in the synthetic 
image. We observe a substantial improvement in matching 
viewed sketches local feature-based discriminant analysis. 
We present a robust method on matching real forensic 
sketches. Using a mug shot gallery of images, we can perform 
race and gender filtering to improve the matching results. The 
results can be then validated by comparing the proposed 
method against existing methods. 


Representation of Sketch to Photo Matching 

The schematic representation of the sketch to photo matching 
system is shown in the following given block diagram i.e. 
figure 1 as follows. 

Here we have a set of sketches (Probe images) and a set of 
gallery photographs to be matched against i.e. mug shot 
photos. 

The steps involved in sketch to photo matching are as follows: 

1 . Store this feature extraction results for every image into a 
feature database. 

2. For every probe image, the corresponding match is that 
with the minimum distance calculated with the nearest 
neighbor matching method. 

3. The final top retrieved images from the database are then 
displayed. 



Figure 1 Schematic representation of the sketch matching 
system 


Feature database is the database maintained where all the results 
or values obtained from the feature extraction method are stored. 
These are afterwards used for matching purpose with the probe 
sketch. Feature value is the calculated feature vector of the given 
input probe sketch image which is to be matched against the 
gallery of mugshot database i.e. the values stored in the feature 
database. 

Matching algorithm is used to find a proper match between the 
probe sketch image with the mugshot images. For example, we 
can match sketch to photos using ‘nearest neighbor matching’ 
method in which the minimum distance between the calculated 
values of the mugshot images and the probe sketch is found 
out. So, this can be used to select the gallery photo with the 
minimum distance to the probe sketch. 

The two phases of the LFDA framework having Training 
phase in the figure 1 and matching phase in figure 2 are 
shown as follows: 

In figure 2 each sketch and photo is represented by SIFT & 
MLBP feature descriptors extracted from overlapping 
patches. After grouping “slices” of patches together into 
feature vectors ® (k) where k=l....N. Then we take a 
discriminant projection ^k for each slice. 
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Fig 2. Overview of Training using the LFDA Framework 


In figure 2 each sketch and photo is represented by SIFT & 
MLBP feature descriptors extracted from overlapping 
patches. After grouping “slices” of patches together into 
feature vectors ® (k) where k=l....N. Then we take a 
discriminant projection ^k for each slice. 

Recognition is performed after combining each projected 
vector slice into a single vector cp and measuring the normed 
distance between a probe sketch and gallery photo. In this 
sketch matching framework, two feature descriptors are used: 
SIFT and LBP. 

The SIFT feature descriptor quantizes both the spatial 
location and gradient orientations within a sxs sized image 
patch, and computes a histogram in which each bin 
corresponds to a combination of a particular spatial location 
and orientation. For each image pixel, the histogram bin 
corresponding to its quantized orientation and location are 
incremented by the product of: 

(i)The magnitude of the image gradient at that pixel . (ii)The 
value of a Gaussian function centered on the patch with a 
standard deviation of s/2. 

Tri-linear interpolation is used on the quantized location of 
the pixel, which addresses image translation noise. The final 
vector of histogram values is normalized to sum to one. It is 
important to reiterate that because we are sampling SIFT 
feature descriptors from a fixed grid, we do not use SIFT 
keypoint detection; the SIFT feature descriptor is computed at 
predetermined locations. 

For the local binary pattern feature descriptor, we can extend 
the LBP to describe the face at multiple scales, by combining 
the LBP descriptors computed with radii r€ {1,3, 5, 7}. This 
is called as the multi-scale local binary pattern (MLBP). 
Local Feature-based Discriminant Analysis 
With both sketches and photos processed using SIFT and 
MLBP image descriptors, we further refine this feature space 
using discriminant analysis. This is done to reduce the large 
dimensionality of the feature vector O. We apply classical 
subspace analysis (such as LDA) directly on ®, and extract 
discriminant features for classification. However, there are 
problems with this approach. First, the feature dimensionality 
is too high for direct subspace analysis. Here each image is 
divided into either 154 overlapping patches (for s = 32) or 720 
overlapping patches (for s = 16), with each patch producing a 
128-dimensional SIFT descriptor or a 23 6 -dimensional 
MLBP descriptor. The second problem is the possibility of 
overffitting due to the small sample size (SSS) problem. 

In order to handle the combination of a large dimensionality 
(feature size) and small sample size, an ensemble of linear 
discriminant classifiers, called local feature- based 


discriminant analysis (LFDA), is proposed. However, we 
choose the proposed LFDA method because it is designed to 
work with a feature descriptor representation as opposed to an 
image pixel representation. 

In LFDA framework each image feature vector ® is first 
divided into “slices” of smaller dimensionality, where slices 
correspond to the concatenation of feature descriptor vectors 
from each column of image patches. Next, discriminant 
analysis is performed separately on each slice by performing 
the following three steps: PC A, within-class whitening, and 
between-class discriminant analysis. Finally, PC A is applied 
to the new feature vector to remove redundant information 
among the feature slices to extract the final feature vector. 
Experiments and results 

A database of 264 sketch-photo pairs is made consisting of 52 
forensic sketches. These images were collected from different 
sources. The experiments are performed using the 
combination of viewed sketches and forensic sketches to 
increase the size of dataset. The database consists of 142 
viewed sketch-photo pairs from CUHK database [2] and 70 
viewed sketch-photo pairs from IIIT-D database [9]. Forensic 
pairs are collected as 25 pairs from Forensic composite sketch 
database [10], which contains sketch-photo pairs from L. 
Gibson [11] and 27 pairs are taken from IIIT-D forensic 
database. 

Initially training was performed on all the sketches with its 
corresponding photographs. And the probe set consisting of 
52 forensic sketches were used to match against a gallery of 
264 gallery images. Matching forensic sketches to large mug 
shot galleries is different in several respects from traditional 
face identification techniques. Hence, when matching 
forensic sketches we are generally concerned with the 
accuracy at rank-50 i.e. whether or not the true subject is 
present within the top-50 images that were near (Euclidean 
distance between descriptors) or top-50 retrieved images. 
Hence keeping in mind that the proposed approach consisting 
of pre-processing method which is only used to improve the 
bad quality of forensic sketches instead of viewed sketches, 
the results with 52 probe set of forensic sketches are obtained 
as shown below. The results are compared with LFDA 
framework and also with SIFT and LBP. 

Examples of the forensic sketches correctly identified at 
rank-1 with both methods are shown in Fig. 3. These sketch 
does a good quality sketch resemble perfectly with the 
suspects photo. 
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Fig 4 shows the worst match which is retrieved at rank 142. 
Comparison with other methods: 

Comparison of all the other methods with the proposed 
method at Rank-50 accuracy is shown as follows in Table 1. 


Methods 

Rank-50 (% Accuracy) 

LFDA 

55 . 76 % 

SIFT 

51 . 92 % 

LBP 

50 . 00 % 


The CMC curve for comparison between the proposed 
approach and other two methods i.e. Scale invariant feature 
transform (SIFT) and Local binary pattern (LBP) is also 
shown as given below in Fig.5.From the CMC curve it can be 
shown that how the rank-50 accuracy of proposed system is 
better than the previous methods i.e. sift and LBP. 



Fig. 5 Rank curve showing comparison between different 
methods 

Performance Evaluation 

The performance of the sketch to photo matching system can 
be evaluated on the following basis given as follow: 
Accuracy: 

Accuracy is calculated as how accurately the correct photo is 
retrieved at the top-most position when a particular sketch is 
given as input. 

Accuracy is calculated as follows 
Rank= Number of correct matches *100 
Total number of input sketches 
In forensic sketch matching, we are matching a sketch to a 
photo, and that sketch too is drawn just based on the verbal 
description of an eye-witness; hence, there are a lot of chances 
for ambiguity. So the law enforcement officers are generally 
concerned with the top P retrieved results. Here, we take P to 
be 50. Hence, number of correct matches is considered from 
top-50 retrieved images. 

Rank 1 Accuracy 


Rank-1 accuracy was calculated using a combined database 
having 264 sketch-photo pairs. 264 sketch-photo pairs were 
used for training purpose. For testing 52 forensic sketch 
images were used. Rank-1 accuracy is defined as the top 
retrieved photo image of the suspect at the Rank-1 position. 
The following are the examples given for the rank-1 retrieved 
photo for the corresponding sketches in Fig 6 


query image Rank = 1 



Figure 6. Correct Rank-1 Matches 

These are typical cases in which the true subject photo was not 
retrieved at rank 1 , but the impostor subject retrieved at rank 1 
looks more similar to the sketch than the true subject as shown 
in Fig 7. 


query image Rank = 1 



Probe Sketch Top Retrieval 



True Suspect 

Figure 7.: Incorrect Rank-1 match 


The top retrieval may sometimes, look visually more similar 
to sketch rather than the true subject. This gives us another 
reason to explain why we consider top-50 retrieved images 
rather than one single image that appears at rank-1. 

V. CONCLUSION 

We performed experiments for matching forensic sketches to 
mugshot photos using a robust feature based method 
LFDA.Matching forensic sketches is a very difficult problem 
in heterogeneous face recognition for two main reasons. 
(l)Forensic sketches are often an incomplete portrayal of the 
subject's face. (2) We must match across image modalities 
since the gallery images are photographs and the probe 
images are sketches. Forensic sketches are drawn by 
interviewing a witness to gain a description of the suspect. 
Research on sketch to photo matching to this point has 
primarily focused on matching viewed sketches despite the 
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fact that real-world scenarios only involve forensic sketches. 
Forensic sketches pose additional challenges due to the 
inability of a witness to exactly remember the appearance of a 
suspect and her subjective account of the description, which 
often results in inaccurate and incomplete forensic sketches. 
Comprehensive analysis, including comparison with different 
methods is performed using the viewed, semi-forensic, and 
forensic sketch databases. Using a collection of 50 forensic 
sketches, we performed matching against a gallery of 264 
images. The results with LFDA method show the rank 50 
accuracy at 55.76%. Thus, the results show that the proposed 
approach is better than other methods such as only using SIFT 
and MLBP. There is a continual research taking place for 
matching forensic sketches. In future a larger collection of 
forensic sketches needs to be collected to further understand 
the complexity of the problem. We observed an improvement 
in matching viewed sketches over published algorithms using 
the proposed local feature -based discriminant analysis. We 
presented a robust method on matching real forensic sketches. 
Using a gallery of images, we can also perform race and 
gender filtering to improve the matching results in future. The 
results can be then validated by comparing the proposed 
method against existing methods. 
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